{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d638561f",
   "metadata": {},
   "source": [
    "# Tool Output Utilization Evaluator\n",
    "\n",
    "### Getting Started\n",
    "This sample demonstrates how to use tool output utilization evaluator on agent data. The supported input formats include:\n",
    "- simple data such as strings and `dict` describing agent responses;\n",
    "- user-agent conversations in the form of list of agent messages. \n",
    "\n",
    "Before you begin:\n",
    "```bash\n",
    "pip install azure-ai-evaluation\n",
    "```\n",
    "Set these environment variables with your own values:\n",
    "1) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
    "2) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
    "3) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
    "4) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41d3647c",
   "metadata": {},
   "source": [
    "The Tool Output Utilization Evaluator assesses how effectively an AI agent utilizes the outputs from tools and whether it accurately incorporates this information into its responses.\n",
    "\n",
    "Scoring is based on two levels:\n",
    "1. Pass - The agent effectively utilizes tool outputs and accurately incorporates the information into its response.\n",
    "2. Fail - The agent fails to properly utilize tool outputs or incorrectly incorporates the information into its response.\n",
    "\n",
    "The evaluation includes the score, a brief explanation, and a final pass/fail result.\n",
    "\n",
    "This evaluation focuses on measuring whether the agent properly uses the information returned by tools to provide accurate and complete responses to users."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "246c7bcd",
   "metadata": {},
   "source": [
    "Tool Output Utilization requires following input:\n",
    "- Query - This can be a single query or a list of messages(conversation history with agent). The original task request from the user.\n",
    "- Response - Response from Agent (or any GenAI App). This can be a single text response or a list of messages generated as part of Agent Response.\n",
    "- Tool Definitions - Tool(s) definition used by Agent to answer the query. Required to understand the context and expected outputs of the tools.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d386d576",
   "metadata": {},
   "source": [
    "### Initialize Tool Output Utilization Evaluator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "087dd96d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.evaluation._evaluators._tool_output_utilization import _ToolOutputUtilizationEvaluator\n",
    "from azure.ai.evaluation import AzureOpenAIModelConfiguration\n",
    "from pprint import pprint\n",
    "\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
    ")\n",
    "\n",
    "\n",
    "tool_output_utilization = _ToolOutputUtilizationEvaluator(model_config=model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7b4f98a",
   "metadata": {},
   "source": [
    "### Samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ee00e9d",
   "metadata": {},
   "source": [
    "#### Evaluating Good Tool Output Utilization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "017a7afe",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle?\"\n",
    "response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_weather_123\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"Seattle\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"tool_call_id\": \"call_weather_123\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"15°C\", \"condition\": \"partly cloudy\", \"humidity\": \"68%\", \"wind\": \"8 mph NW\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"The current weather in Seattle is partly cloudy with a temperature of 15°C. The humidity is at 68% and there's a light wind from the northwest at 8 mph.\",\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "result = tool_output_utilization(query=query, response=response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78270b9b",
   "metadata": {},
   "source": [
    "#### Complex Scenario with Multiple Tool Outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8543c52",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Can you send me an email with weather information for Seattle?\"\n",
    "response = [\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:35Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"Seattle\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:37Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"14°C\", \"condition\": \"rainy\", \"precipitation\": \"light rain\", \"wind\": \"12 mph SW\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:38Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
    "                \"name\": \"send_email\",\n",
    "                \"arguments\": {\n",
    "                    \"recipient\": \"your_email@example.com\",\n",
    "                    \"subject\": \"Weather Information for Seattle\",\n",
    "                    \"body\": \"Current weather in Seattle: 14°C and rainy with light rain. Wind is from the southwest at 12 mph. Stay dry!\",\n",
    "                },\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:41Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [\n",
    "            {\"type\": \"tool_result\", \"tool_result\": {\"message\": \"Email successfully sent to your_email@example.com.\", \"status\": \"delivered\"}}\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:42Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"I have successfully sent you an email with the weather information for Seattle. The email includes the current conditions: 14°C and rainy with light rain, plus wind details. The email was delivered successfully to your inbox.\",\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "result = tool_output_utilization(query=query, response=response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c52f24c",
   "metadata": {},
   "source": [
    "#### Query as Conversation History"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3b9bef5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Query as conversation history instead of a single string\n",
    "query_as_conversation = [\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": \"You are a helpful assistant that can fetch weather information and send emails.\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\", \n",
    "        \"content\": \"Hi, I need to know the weather in Seattle for my trip planning.\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": \"Can you also email me a summary so I can reference it later when packing?\"\n",
    "    }\n",
    "]\n",
    "\n",
    "response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_weather_123\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"Seattle\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"tool_call_id\": \"call_weather_123\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"15°C\", \"condition\": \"partly cloudy\", \"humidity\": \"68%\", \"wind\": \"8 mph NW\", \"forecast\": \"no rain expected\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_email_456\",\n",
    "                \"name\": \"send_email\",\n",
    "                \"arguments\": {\n",
    "                    \"recipient\": \"user@example.com\",\n",
    "                    \"subject\": \"Seattle Weather Summary for Trip Planning\",\n",
    "                    \"body\": \"Weather summary for Seattle: Temperature is 15°C, partly cloudy conditions, humidity at 68%, with northwest winds at 8 mph. No rain is expected, so light layers should be sufficient for your trip.\"\n",
    "                },\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"tool_call_id\": \"call_email_456\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"message\": \"Email sent successfully\", \"delivery_status\": \"delivered\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"Perfect! I've checked the weather in Seattle for your trip planning. It's currently 15°C and partly cloudy with light northwest winds and no rain expected. I've also sent you an email summary with all these details including packing recommendations based on the weather conditions. The email was delivered successfully, so you can reference it when preparing for your trip.\",\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "result = tool_output_utilization(query=query_as_conversation, response=response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "199a11bc",
   "metadata": {},
   "source": [
    "#### Example of Poor Tool Output Utilization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6be39269",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle?\"\n",
    "# Poor utilization - agent doesn't use the detailed tool output effectively\n",
    "poor_response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_weather_123\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"Seattle\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"tool_call_id\": \"call_weather_123\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"15°C\", \"condition\": \"partly cloudy\", \"humidity\": \"68%\", \"wind\": \"8 mph NW\", \"UV_index\": \"3\", \"visibility\": \"10 km\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"The weather in Seattle is okay. It's not too bad outside.\",  # Poor utilization of detailed weather data\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "# This should score poorly due to inadequate use of tool output\n",
    "result = tool_output_utilization(query=query, response=poor_response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcb8a2cf",
   "metadata": {},
   "source": [
    "#### Response as String (str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0845822f",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What's the weather like in Seattle?\"\n",
    "# Response as a simple string (not a list of messages)\n",
    "response_str = \"The weather in Seattle is currently 15°C and partly cloudy with light winds from the northwest.\"\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "result = tool_output_utilization(query=query, response=response_str, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4cb9ddd",
   "metadata": {},
   "source": [
    "#### Tool Definitions as Single Dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6424b3bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Check the weather in Seattle\"\n",
    "response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_weather_456\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"Seattle\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"tool_call_id\": \"call_weather_456\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"temperature\": \"14°C\", \"condition\": \"rainy\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"The current weather in Seattle is rainy with a temperature of 14°C.\",\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "# Tool definition as a single dict (not a list)\n",
    "tool_definition_dict = {\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "    },\n",
    "}\n",
    "\n",
    "result = tool_output_utilization(query=query, response=response, tool_definitions=tool_definition_dict)\n",
    "pprint(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_agent_evaluator_prp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
